feat: add Google Cloud Build support for harness-config image builds#521
Conversation
Add CloudBuildHarnessConfigExecutor that builds harness-config images via Google Cloud Build API instead of local Docker/Podman. The executor uploads build context to GCS, submits a Cloud Build job with multi-arch support (linux/amd64,linux/arm64), and streams logs back to the maintenance run logger. Route to the Cloud Build executor via a "builder" parameter on the existing build-harness-config-image operation: when params["builder"] is "cloud-build", the request delegates to CloudBuildHarnessConfigExecutor. Also adds cloud.google.com/go/cloudbuild/apiv1 dependency and wires GCPProjectID from ServerConfig into the executor.
Support --builder cloud-build to submit harness-config image builds via gcloud builds submit. Resolves GCP project from settings or gcloud config. Cloud Build defaults to multi-arch (linux/amd64,linux/arm64) and requires image_registry to be configured. Refactors config update and Hub sync into a shared helper to avoid duplication between local and Cloud Build paths.
Add cloudBuildAvailable and gcpProjectId fields to OnboardingStatus so the frontend can conditionally show the Cloud Build build option when a GCP project is configured.
- Clean up GCS source tar.gz after build completes (deferred delete) - Check staging bucket exists before upload with actionable error - Deduplicate materializeHarnessConfigFiles: local executor now uses shared helper - Deduplicate syncBuiltImage: extract shared top-level function for both executors - Replace fmt.Sprintf YAML interpolation with yaml.Marshal in CLI cloud build
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for Google Cloud Build as an alternative build backend for building harness-config images, both via the CLI and the Hub maintenance executor. It adds a new --builder flag, implements GCS upload and Cloud Build job submission, and updates the onboarding status to reflect Cloud Build availability. The review feedback highlights a critical compilation error due to a type mismatch in the build operation handling, which can be resolved by using the type-safe op.Metadata() method. Additionally, a cross-platform path separator issue on Windows was identified in the tar archiving logic, which should be fixed using filepath.ToSlash.
| buildID, err := extractBuildID(op) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to get build ID from operation: %w", err) | ||
| } |
There was a problem hiding this comment.
The code will fail to compile here because op is of type *cloudbuild.CreateBuildOperation (returned by cbClient.CreateBuild), but extractBuildID expects *longrunningpb.Operation.
Instead of manually unmarshaling the metadata from the raw operation, you can use the type-safe Metadata() method provided by the CreateBuildOperation wrapper. This also allows you to completely remove the helper function extractBuildID and its associated imports (proto and longrunningpb).
| buildID, err := extractBuildID(op) | |
| if err != nil { | |
| return fmt.Errorf("failed to get build ID from operation: %w", err) | |
| } | |
| meta, err := op.Metadata() | |
| if err != nil { | |
| return fmt.Errorf("failed to get build metadata: %w", err) | |
| } | |
| if meta.Build == nil || meta.Build.Id == "" { | |
| return fmt.Errorf("build metadata does not contain a build ID") | |
| } | |
| buildID := meta.Build.Id |
| if err != nil { | ||
| return err | ||
| } | ||
| header.Name = relPath |
There was a problem hiding this comment.
On Windows, filepath.Rel returns paths with backslashes (\\). If the tar header name contains backslashes, Google Cloud Build (which runs on Linux) will extract them as flat files with literal backslashes in their names instead of creating the expected directory structure. This will cause the build to fail (e.g., Dockerfile not found).
Using filepath.ToSlash ensures that forward slashes are always used as path separators in the tar archive, making it cross-platform compatible.
| header.Name = relPath | |
| header.Name = filepath.ToSlash(relPath) |
| // extractBuildID extracts the Cloud Build ID from a CreateBuild long-running | ||
| // operation response. | ||
| func extractBuildID(op *longrunningpb.Operation) (string, error) { | ||
| if op.Metadata == nil { | ||
| return "", fmt.Errorf("operation has no metadata") | ||
| } | ||
| var meta cloudbuildpb.BuildOperationMetadata | ||
| if err := proto.Unmarshal(op.Metadata.Value, &meta); err != nil { | ||
| return "", fmt.Errorf("failed to unmarshal build operation metadata: %w", err) | ||
| } | ||
| if meta.Build == nil || meta.Build.Id == "" { | ||
| return "", fmt.Errorf("build metadata does not contain a build ID") | ||
| } | ||
| return meta.Build.Id, nil | ||
| } |
There was a problem hiding this comment.
Since we are now using op.Metadata() to retrieve the build ID directly, this helper function is no longer needed and can be safely removed.
filepath.Rel returns OS-specific separators. On Windows this produces backslashes in tar headers, breaking Cloud Build (Linux) extraction.
1 participant
Summary
Adds Google Cloud Build as an alternative backend for building harness-config container images, alongside existing local Docker/Podman/Apple Container support.
CloudBuildHarnessConfigExecutorthat uploads build context to GCS, submits builds via the Cloud Build Go API, and streams logs back through the existing maintenance run polling. Activated viabuilder: "cloud-build"param on the existingbuild-harness-config-imageoperation.scion build <harness-config> --builder cloud-buildgenerates a temporary cloudbuild.yaml and submits viagcloud builds submit. Requires gcloud CLI, a configured registry, and a GCP project.cloudBuildAvailableandgcpProjectIdfields for frontend feature gating.linux/amd64,linux/arm64via buildx, a key advantage over local builds.materializeHarnessConfigFiles()andsyncBuiltImage()helpers deduplicate code between local and Cloud Build executors.Test plan
builderparam omitted (backward compat)scion build <hc> --builder cloud-buildwith a GCP project that has Cloud Build API enabled{"params": {"harness_config_id": "...", "builder": "cloud-build"}}cloudBuildAvailableis true/false based ongcp_project_idconfig